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A uniformly distributed (iid) binary source is encoded into two 
binary data streams at rates Ri and R2, respectively. These sequences 
are such that by observing either one separately, a decoder can 
recover a good approximation of the source (at average error rates 
D\, D2, respectively), and by observing both sequences, a decoder can 
obtain a better approximation of the source (at average error rate 
Do). In this paper a "converse" theorem is established on the set of 
achievable quintuples (R\, R2, Do, D h Do). For the special case Ri = 
R2 = 1/2, Do = 0, and D\ = D2 = D, our result implies that D > 1/5. 

I. INTRODUCTION 

Let {Xk}k-i be a sequence of independent drawings of the binary 
random variable X, where Pr{X = 0} = Pr{X = 1} = 1/2. Assume that 
this sequence appears at a rate of 1 symbol per second as the output 
of a data source. (Refer to Fig. 1.) An encoder observes this sequence 
and emits two binary sequences at rates JBi, Ro ^ 1. These sequences 
are such that by observing either one, a decoder can recover a good 
approximation to the source output, and by observing both sequences, 
a decoder can obtain a better approximation to the source output. 
Letting D\, Do, and Do be the error rates which result when the streams 
at rate R u rate Ro, and both streams are used by a decoder, respec- 
tively, our problem is to determine (in the usual Shannon sense) the 
set of achievable quintuples (R h R 2 , Do, D\, D 2 ). Our main result is a 
"converse" theorem which gives a necessary condition on the achiev- 
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Fig. 1 — Communication system. 

able quintuples. This paper extends a previous one on the same sub- 
ject. 1 This paper, however, is self-contained. 

This problem is an idealization of the situation in which it is desired 
to 

(i) send information over two separate channels, as in a packet 
communication network, and 

(ii) recover as much of the original information as possible, should 
one of the channels break down. 

To fix ideas, let us say that Ri = R 2 = 1/2, Do = 0, and Di = D 2 = D. 
Thus, the source sequence at rate 1 is to be encoded into two sequences 
of rate 1/2 each, such that the original sequence can be recovered from 
these two encoded sequences with approximately zero error rate (i.e. 
D = 0). Our question then becomes: How well can we reconstruct the 
source sequence from one of the encoded streams — that is, what is the 
minimum Dl A simple-minded approach would be to let the encoded 
streams consist of alternate source symbols, which will allow Do = 0. 
In this case, D = 1/4, since by observing every other source symbol a 
decoder will make an error half the time on the missing symbol. Is it 
possible to do better? El Gamal and Cover 2 have looked at this problem 
and have a theorem which can be used to show that we can make 
D = (-v/2 - l)/2 = 0.207. In a previous paper 1 it was shown that (with 
R 1 = R 2 = 1/2, Do = 0) D > 1/6. The new result given here specializes 
to D > 1/5 = 0.200. The exact determination of the best D remains 
an open problem.* 

II. FORMAL STATEMENT OF PROBLEM AND RESULTS 

Let B = {0, 1}, and let d«(x, y), x, y e B N , be the Hamming distance 
between the binary N- vectors x, y; i.e., cfo(x, y) is the number of 
positions in which x and y do not agree. A code with parameters 



* In Ref. 3, Witsenhausen proved a closely related result which encourages the 
conjecture that D = 0.207 is, in fact, the best possible. 
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(N, M h M 2 , Do, D\, D 2 ) is a quintuple of mappings (f, f 2 , go, gi, g 2 ) 
where, 

f a :B N ^ {1, ... ,M a ),a = 1,2 (la) 

g a :{l,2, • •• ,M a }^B N ,a=l,2 (lb) 

go:{l ) 2,... I M 1 }x{l ) 2,...,M 2 }^B Jv . (lc) 

The source output is a random vector X uniformly distributed on B^. 
Define 

Y = £ l0 /i(X), (2a) 

Z = g 2 °f 2 (X), (2b) 

and 

X = £o[/i(X),/ 2 (X)]. (2c) 

Then the average error rates are 

D,=^Ed H (X,Y), (3a) 

D 2 = ±-Ed H (X,Z), (3b) 



N 



D = ±Ed„(X,X). 



(3c) 



We say that a quintuple (R u R 2 , do, d\, d 2 ) is achievable if, for 
arbitrary e > 0, there exists, for N sufficiently large, a code with 
parameters (N, M u M 2 , Do, D u D 2 ), where M a < 2 lRa+e)N , a = 1, 2, and 
D a < d a + e, a = 0, 1, 2. The relationship of this formalism to the 
system of Fig. 1 should be clear. Our problem is the determination of 
the set of achievable quintuples, and our main result is a converse 
theorem. 

Before stating our result, let us take a moment to state a positive 
theorem by El Gamal and Cover 2 as it specializes to our problem. 

Theorem 1: The quintuple (Ri, R 2 , do, d\, d 2 ) is achievable if there 
exists a quadruple of random variables X, X, Y, Z, which take values 
in B, such that Pr{X = 0} = Pr{X = 1} = 1/2, and 

Ed H (X, X)<d , (4a) 

Ed„(X,Y)^d u (4b) 

Ed H (X,Z)<d 2 , (4c) 

and 

R^KX-Y), (5a) 
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R 2 >I(X;Z), (5b) 

Ri + R 2 > I(X; X, Y, Z) + I(Y; Z) , (5c) 

where /(•;•) is the usual Shannon information. 

For the special case of Ri = R 2 = 1/2, do = 0, it can be shown that 
d x = d 2 = (v2 - l)/2 a 0.207 is achievable. 

We now state our converse result. 
Theorem 2: If (R u #2, d , d u d 2 ) is achievable, then in all cases 

R 1 + R 2 >l-h(d ), (6a) 

furthermore, if2di + d 2 < 1, then 

(2d 2 \ 
2<f 1 + c? 2 - ' 1 , (6b) 



and if d\ + 2d 2 < 1, then 



2d? 



Ri + R 2 >2- h(do) -h[dx + H-jT^I. (6c) 



where 



(0, A = 0, 

/i(X) = J -X l og2 X - (1 - A)log 2 (l - A), < A < 1/2. 
I 1, * s V2. 

All logarithms in this paper are taken to the base 2. As (6a) is obvious, 
and (6c) follows from (6b) by symmetry, we need only prove (6b). 

In the special case of Ri = R 2 = 1/2, do = 0, and di = d 2 = d, 
inequality (6b) implies that 

/ 2d 2 \ 

or 

2d 2 _ 3d(l -d)- 2d 2 1 
M 0^T)~ d-d) ~2' 

which implies that d > 1/5 = 0.200. 

III. PROOF OF THEOREM 2 

We start from the standard identity 

I(Ui; U 2 Us) =I(Ux; U 2 ) + I(Ui] U 3 \ U 2 ), (7) 

for arbitrary random variables Ui, U 2 , U* We say that Ui, U 2 , U 3 is a 
"Markov chain" if Ui, U 3 are conditionally independent given U 2 \ i.e., 
Us depends on U x , U 2 only through U 2 . If Ui, U 2 , Us is a Markov chain 
then I(Uu U 3 \ U 2 ) = 0, and from (7) 
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/(£/,; Us) < I(Ui; U 2 U 3 ) = I(Uu U 2 ). (8) 

Note that the hypothesis for (8) holds when U 3 is a function of U2. A 
sequence {U„} is a Markov chain if, for all n, 

(• • • Un-2, U„-i), U„, (U n +h U n +2, • • ') 

is a Markov chain. 

Let us now suppose that we are given a code (/!, f 2 , go, g\, ^2) with 
parameters (N, Mi, M 2 , Do, D\, D 2 ). We can write 

log Af, + log M 2 > H(MX)) + H(f 2 (X)) (9) 

= I(fi(X); f 2 (X)) + H(fAX)f 2 (X)) 

= I(fi(X); f 2 (X)) + I(X; fi(X)f 2 (X)) (10) 

> I(fi(X); f 2 (X)) + J(X; X, Y, Z), (11) 

where (9) follows from the fact that /}(X) takes its values in a set of 
cardinality Mi, (10) holds because the pair /i(X) /2(X) is determined by 
X and (11) holds because X, Y, and Z depend on X only through /i(X) 
/2(X) so that (8) applies. 

Now (11) is getting close to (5c) in the direct theorem. 
In fact, using (8) twice, we can underbound J[/i(X); /2(X)] by 
/(Y; Z). Now the components of the source vector X are independent, 
and if the components of either Y or Z were also independent, we 
could make use of standard techniques to establish the necessity of 
(5c). But alas, we cannot assume that either the {Y n ) nor the {Z n ) are 
independent, so that another tactic is required. The key idea is the 
definition of another random vector V = (Vi, • • • , V n ) the components 
of which are in fact independent. 

For 1 < k < Mi, define the set 

A k ={K:fi(^)=k)=fT 1 (k). (12) 

Let the cardinality of A k be u*. Let the random vector V be defined by 
its conditional distribution given X: 

V6A /l(x) , 

otherwise . 



Pr{V = v|X = x} = |^ ,r ' 



Thus, given X € A k , V is uniformly distributed on Ak. It follows that 
the unconditional distribution for V is 

p r {V = v} = 2~ n , v e B", 

and the components of V are independent, as desired.* Furthermore, 
Z, /i>(X), X, /i(X), V is a Markov chain, so that, using (8), 



* In effect, V is obtained from f x (X) by a channel with transition probabilities 
Pr{X = xl/i(X) = k) so that the distribution of V is the same as that of X, hence, iid. 
This is valid for any distribution of X. 
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7(/i(X); f 2 (X)) = J(V, A(X); / 2 (X), Z) > 7{V; Z). 
Combining (11) and (14), we obtain 

i log Mi + ilogM 2 >i/(V; Z) + ^ J(X; £, Y, Z) 



(14) 



^ 7(V;Z)+ ^ /(X; * ) 



CD 

> 2-h 



Ed H (V; Z) 



2V 



-h 



Ed„(X, &) 



where 



= 2 - /i(A) - /i(Z)o), 
£cfe(V; Z) 



A = 



N 



(15a) 



(15b) 



Step (1) follows from the "rate-distortion bound" which states that 
if U is a random vector uniformly distributed in B N (as are V and 
X), and f)" is an arbitrary binary random vector, then 7(U; U) > 

1-fti Ed„(U, ft) . (See Ref. 4.) 

_N 

We will now obtain an upper bound on A in terms of A and D 2 . As 
a "warm up," let us observe that from the triangle inequality, 

A = -J- Ed H (V, Z) < -J- [Ed„(V, Y) + Ed H (Y, X) + Ed„(X, Z)]. 
AT N 

Now 

£rf//(Y, X) = D,N, Ed H (Z, X) = AN, (16) 

Furthermore, 

Ed H (V, Y) = £ £[d„(v, Y) | V = v] Pr{V = v}. 

V 

Now suppose that we are given V = v e A*. Then, Y = g\(k). Since 
p r {V = v} = 2~ N , 

M, 

-l-N. 



£cMV, Y) = £ S 2- N d/f[v, ^i(A)] 



M, 



= E E PKX = *}d H [x, gx(k)] = NDx. 



(17) 



Thus, 



A < 2 A + A- (18) 
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Substitution of (18) into (15a) yields that for achievable 
(#,, R 2 , do, d u d 2 ) 

R i + R 2 >2- h(2di + d 2 ) - h(d Q ), (19) 

which is the result reported in Ref. 1. 

We will now establish a tighter bound on A, namely, for D 2 + 
2Di < 1, 

1 2D 2 

A = -Ed„(\, Z) < D 2 + 2A '—, (20) 

N (1 - D 2 ) 

so that (15) yields that for achievable (Ri, R 2 , do, d\, d 2 ), 

R x + R 2 >2- h(d 2 + 2di - yztt) ~ &<*>> ( 21 > 

which is (6b), the inequality required for Theorem 2. 

Upper bound on A: We establish inequality (20) as follows. Let k, 
\<k<Mihe fixed. Let A k be as defined in (12), and let its cardinality 
Hk = /a. Let the members of A* be the N- vectors (Xm}m-i- Let y = g\{k). 
Thus, when X = x e A k , then Y = y. Now, form a n X N 
array, A, with /nth row 

a m = (a m „ a m2 , • • • , a mN ) = x™ © y, (22) 

where © denotes modulo 2 vector addition. Thus, a mn = 1, when the 
nth coordinates of x„, and y are different, and a mn = 0, otherwise. Note 
that 

±E[d Il (X,Y)\f i (X)=k] 

1 1 £ 1 1 M N 

= t; - 2 ^«( x ">, y) = t; - S X a M „ 

1 " 
= tj I *«> (23a) 



where for 1 < n < AT, 



1 M 
s„ = - J a m „ (23b) 



is the fraction of l's in column n of A. 

Next, for 1 < m < /x, let z m = g 2 °f 2 (x. m ) be the value of Z which 
results when X = x m . Let B be the n X N array with mth row 

b m = (6mi, b m2 , • • • , b mN ) = z m © y. (24) 
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Then, 
±E[d H (X,Z)\MX) = k] 



1 1 M 



= ttS d H (a m , b m ) 

- T? S I" 2 [ft""* 1 " 6 -") + W 1 " *■»)] f' (25> 

iV „=i ^/l m =i 

where the last equality foUows from the fact that for a, 6e{0,l}, 
a(l — 6) + 6(1 — a) = or 1 according as a = b or a ^ b. Let t„ be the 
expression in braces in (25). Then, 
1 M 

Tn = ~ A [flm/i + b mn — 2a mn 0m/iJ 



- Y [a mn — b mn + 2b mn (l — a mn )] 

it m 

— ~~ y, \0-mn Omn) = Sn wi 

- y. [b mn — a mn + 2a m „(l — b mn )] 

ilm 



where 



— 7. [Omn &mn) *n »rt| 
M m 



tn - " I 6mn, 1 < 71 < N, 

M 771=1 



(26a) 



(26b) 



and s n is given by (23b). We conclude that t„ > \t n — s„\, so that 
(25) yields 

i £[d w (X, z) | A(X) - A] > i £ |*„ - s„j. 



2V 
Finally, consider 

±E[d H (V,Z)\MX) = k] 



N n % 



(27) 



-t; 2 i^[d«(V,Z)|X = x m ] 
= W Z E[d H (V,z m )\X = x m ] 
= ^ZZ d w (v, zj Pr{V = v|X - x-}. (28) 

iVll m=l v 
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Now, from (13) which defines V, 



J " J I 0, otherwise. 



Thus, (28) yields, 
i^[rfjf(V,Z)|/i(X)-A] 



2E E rf/f(x m -, z m ) 



1 



1 



2 rf//(a m -, b m ) 



X 2 [o/n-nU — 6 mn ) + (1 — a mn )6 mn ] 



*'/t m,m' n 

= 4 E [*»(i - ftO + Mi - *)]. (29) 

Now make the dependence of s„ and £„ on k explicit by writing s„* 
and tnk> respectively. Then, on averaging over k, (23b) becomes 

1 Mi 1 N 

D 1 = -E{d„(X,Y)} = I Pr{/i(X) = *}- I s„* 

iV k-l iV n _i 

Af, A/ 

= 2 2 P(#i,A)s„ A , (30a) 

where P(n,k) = Pr{/i(X) = k) X—. Similarly, (27) becomes 

, Af, n 

D 2 = e {rf H (X, Z)} > £ J P(n, A) I ft* - »„* I . (30b) 

iV k-l n-l 

Finally, (29) becomes 

b = ±E{d„(V t Z)} 

M, AT 
= J £ ^fo *)[»n*(l - ***) + tnkd ~ 8nk)l (30c) 

We now apply the following inequality, the proof of which is given 
in Section IV: 

Let S, T be random variables such that < S, T < 1, and E {S} < 
D U E{\T-S\)<D 2 , with 2ft + ft < 1. Then, 

2D? 

£{S(1- T) + T(l-S)} <ft + 2A-- J-. (31) 

1 — ft 
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Let S, T be the random variables which take the value s„*, tnk, 
respectively, with probability P(n, k), then (30) and (31) imply that, 
for 2A + A<1, 

2D\ 



A < A + 2 A - 



(32) 



1-A' 

which, when substituted in (15) gives (6b), proving Theorem 2. 

IV. PROOF OF THE INEQUALITY 

Define Q(A, A) as the supremum of E{S(1 - T) + 7(1 - S)} 
over all distributions of (S, T) on the unit square [0, l] 2 for which 
E{S] < A, E{\T - S\) s A. 

Theorem 3: (a) For 2A + A < 1 owe /ias 

2D? 



Q(A,A) = 2A + A- 



1- A' 



u;^ Q{0, 1) = 1. (b) For 2A + A > 1 one Aas 

Q(A,A)>i. 

To establish this, introduce for S, T, x, y in [0, 1], v ^ 1, the 
function * 



F(S, T, x, y) = 5(1 - T) + T(l - S) 



4* 



1-v 



- 2 (S - *) + 



2x 5 



(1 - y)' 



- 1 



(|S-T|-v). (33) 



Lemma: For 2x + y < 1, y < 1 tfie maximum ofF(S, T, x, y) over all 

(S, T) in [0, l] 2 is 2x + y - 2* 2 /(l - y). 

Proof: For fixed S, the maximum of F over T must, by piecewise 

linearity, be at either T = 0, S, or 1. 

(*") IfT=0, then 



F=S + 



4x 



1-v 



-2 (S-*) + 



2x' 



(1 - y)' 



-1 



(S-y), 



and this is maximized over S at either S = or S =1. 
(a) For 5 = 

r, « 4x 2 2x 2 v 

F = 2x - + y- 



1-y 
= 2x + y- 



2x' 



(1 - y) 2 
2x 2 2x 2 y 



1-y 1-y (1-y) 



< 2x + y - 



2x< 



i-y 



* This choice of F comes from the duality theory of convex hull formation, as 
described, e.g., in Section VA of Ref. 5. 
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(b) For S = 1 

4x(l-x) 2x 2 
F=l-2 + 2x+ — + 1 + y 

i-y i- y 

2x 2 4x 2x 2 
= 2x + v - 2 + < 2x + y - , using 2x + y < 1. 

i -y i -y i-y 

(ii) If T = S, then 

F = 2.S :>S : ' I ( t— -2)(S-x) + y- ''* y 



i-y ~) ,_ ""' " (l-yr 

The maximum over S of this quadratic is at S = x/(l — y) which is in 
[0, 1] if x < 1 - y and this holds as x < (1 - y)/2 < 1 - y. Hence, the 
maximum is 

2x 2 



2x 



i-y (l-yr U -y 



+ ,^L-2 



— x 



l-v 



2x 2 y „ 2x 2 

+ y-- — ^72 = 2* + ^" 



(m) KT= 1, then 



F= 1-S+ 



4* 



l-v 



- 2 ](S - x) + 



(1 - yY 

2x 2 



(l - y\ 



which is maximized by taking S either or 1. 
(a) ForS = 

4x 2 2x 2 

F = 2x + y - + 



- 1 



2x 2 y 



1-y 
(l-S-y), 



= 2x + y - 



(b) For S - 1, 



F = 2x + y + 

= 2x + v - 
< 2x + y - 



4x 



i-y ' (i-y) 2 (1-y)' 

l-v 



4* 2 2x a y 



1-^ l-v 
2x 2 



(l - y)' 



2a: 2 4x „ 2x 2 y 
+ 2 - 



1-y 1-y 1-y 
2x 2 



(i-y)' 



i-y 



since 4x/ (1 - ,y) < 2. 

Thus, the maximum is as stated and is attained for T = S = x/ 
(l-v) and for T = 1, S = 0. This completes the proof of the lemma. 

Turning to the proof of Theorem 3, consider any distribution of 
(S, T) on the unit square for which 
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E{S)=x, E{\T-S\} = y, 2x + y<l. (34) 

If y = 1, then x = from which it follows that S = 0, T = 1 almost 
surely, giving E{S{1 - T) + T{1 - S)} = 1. 
If y < 1, one has, by the lemma, 

Or 2 

E(F(S, T, x, y)} = E{S(l - T) + T(l - S)} < 2x + y-- . 

i - y 

If one chooses the distribution T = 1, S = with probability y and 
T = S = x/(l — y) with probability 1 — y, equality is attained. This 
determines the maximum of E{S(1 - T) + T(l - S)} subject to (34). 
As 2x + y — 2:r 2 /(l — y) is monotone increasing (for 2x + y < 1) in 
both x and y, the maximum is unchanged if one allows all x, y with 
< x < D\, < y < D 2 . This establishes part (a) of Theorem 3. 

For part (b), it suffices to observe that Q is monotone nondecreasing 
in both arguments by its definition and that on the boundary, where 
2Dy + D 2 = 1, one has Q(D U Do) = (1 + D 2 )/2. This establishes part 
(b). (It could easily be shown that Q(Di, D 2 ) = (1 + D 2 )/2 for all 
(Di, D 2 ) in the unit square satisfying 2D\ + D 2 > 1.) 
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